Data Driven Gait Analysis

Classification of Cross-Country Skiing Techniques Using Supervised Learning

TRA105 Digitalization in Sports | Chalmers University of Technology

Group Members:



In this notebook, we will be looking build a Random Forest Classifier model to identify Gait in skiers. Further details regarding the setup of these experiments has been provided in their respective sections. Please note that this notebook focuses on only the implementation of classification models and their performance on data. Any information regarding the raw data, pre-processing of raw data, cleaning, preparation etc can be found within the files present in the analysis_setup_functions and dataframe_functions files of this package.

Instructions:

You can understand more about these functions by reading through their files where they have been defined and explained in detail

Contents

You may click on these links to navigate to different sections of the notebook directly.

(These links may not work when viewed on GitHub or Colab directly, you may download and open the file on jupyter or equivalient for this functionality. Alternatively for Colab, you can use the left panel to navigate the sections)

  1. Setting up the Random Forest Classifier Method
  2. Cross Validation of Model
  3. Summary of Cross Validation
  4. Additional Experiments using Model
  5. Observations of Random Forest Classifier

We also perform a refininement of predicted labels using a majority filter technique and evaluate its results by repeating experiments similar to the ones listed above. These can be accessed byy the below links.

  1. Setting up the Refined Results using Majority Filter
  2. Cross Validation and Refining Predicted Labels
  3. Summary of Cross Validation and Comparison of Refined Results
  4. Additional Experiments by Refining Predicted Labels
  5. Observations of Refined Predicted Lables

Skier Profile

Below are the Skier Profiles for the 10 Subjects this data was recorded from:

Skier Gender Mass (kg) Height (cm) Pole Length (cm) Body Mass Index (BMI)
1 Man 87.5 190 167.5 24.2
2 Man 78.5 190 167.5 21.7
3 Man 88.3 190 167.5 24.5
4 Kvinna 63.9 161 147.5 24.7
5 Man 77.6 186 160 22.4
6 Man 69.8 178 160 22.0
7 Kvinna 64 161 147.5 24.7
8 Man 66.6 176 157.5 21.5
9 Man 73.3 186 160 21.2
10 Kvinna 60.5 185 160 17.7



Importing Libraries

Back to Top

Loading the Dataset

Back to Top

We load the entire dataset of all the skiers together, so that we can later split into trianing and test according to our convenience. If you want to change the number of skiers that should be chosen for validation, please change the validation_skiers variable to a number of your choice. We have set it to 3. df_info is the dataframe which contains all the information useful for running our model and that is the one used by us in this notebook.




RANDOM FOREST CLASSIFIER

Back To Top


Now that we have our data loaded from the above code block, we will see how well a random forest classifier method performs in identifying the gears. The execution of our experiment using random forest classifier will be carried out in the following steps:

The experiment is performed within the for loop which is used to repeat the experiments for 10 runs (in our case). We will then view the results across all the runs and see if the performance of this method is consistent, and share our observations.

Hyperparameter Tuning

Feature Importance Score Plot

After tuning the random forest classifier, we also would like to see the feature importance scores assigned to the features of our dataset. However, we will not be performing any feature selection by removing any feature as there are very few features in comparison the the number of samples of data.

Cross Validation on Tuned Model

Back To Top

Summary of RFC Model Across 10 Runs

Back To Top

Additional Experiments on Model

Back To Top

Observations

Back To Top

Observations from Cross Validation:

As we can see, the Random Forest Classifier performs well in predicting the gears across all runs on the validation dataset. Across 10 runs, we get a Mean Accuracy of 0.75 and a Mean Balanced Accuracy of 0.72 with a Mean MCC Score of 0.630 which indicates that this is potentially a good classifier for our data.

The performance of our models in each of the runs varied between 0.64 and 0.82, we have tested this method on different combinations of training and validation sets and can thus be fairly confident that it would perform similarly with somewhat similar performance metrics for any new data we test it on.

In each of the cases, prediction of Gear 4 was poor, it had the lowest precision among all the gears consistently. This could be due to the fact that it has a considerably smaller set of values in our dataset so the model cannot learn more information about it, and balancing the class weights does not completely rectify the same.

Observations from Additional Experiments:

Interestingly, our model seems to perform comparatively well when we use only females as our validation dataset. An accuracy of 0.723 is close to the mean accuracy of the model after cross validation, although slightly on the lower side. The females in this experiment also happen to have the lowest mass, shortest height and shortest pole lengths among the skiers we used. Thus we could say this model preforms comparatively well, with slightly reduced accuracy, for skiers with such profiles when trained on skiers that are males, or tall or have higher mass.

Conversely, however, selecting the tallest skiers for validation yielded an improved performance of the model from the mean CV score. An accuracy of 0.76 is slightly higer than the mean, but if we look at the confusion matrix closely for this experiment, we see that the Gear 4 has been predicted very poorly, and the accuracy is potentially skewed due to the high accuracy of gear 0 and gear 3. This is further evidenced bya low Balanced Accuracy of 0.694and a low F1 Score.

We also tried to see if a combination of mass and height, such as a body mass index (BMI) calculated as (mass (in kg))/(height (in m))^2 would impact the accuracy of prediction. We saw that validating on skiers with higher BMI resulted in poor results.



Refining Predicted Labels using Majority Filter

Back To Top

Now, we explore an additional step where we try to refine the results we have obtained. As we saw in the previous section results, each observation in our dataset is being treated individually and returns a label per observations. However, in some areas, we see that the predicted labels are switching between gears very frequently and such high variation in gear in a small window of observations is not realistically possible, as a skier will not be switching between gears so often.

So, we could potentially further refine our predicted labels by applying a majority filter on the labels. In a majority filter, we define a window size (15 in our case), where for every label, the filter calculates the most commonly occuring label around the current label and updates the current label to that value. This way, if there are outliers within a range of observations, they could be 'corrected' using the majority filter.

This section repeats experiments similar to the above, but in addition also computes the refined results and presents them for side by side comparison. We will discuss our findings at the end of the experiments.

Cross Validation and Refining Predicted Labels

Back To Top

Summary of Cross Validation and Comparison of Refined Predicted Labels

Back To Top

Original Results

Refined Results

Observations From CV and Refining Predicted Labels

It is clear from the above that there is a visible improvement in accuracy of the results when we remove/modify teh outliers within a range of observations for a gear using the majority filter. The Mean accuracy jumps from 0.756 to 0.827 and the balanced accuracy improves from 0.727 to 0.792.

Currently the window width for the filter is 15 neighboring values by little trial and error, but maybe by playing around with that width parameter further, we could increase the performance further.

Refined Results for Additional Expermients

Back To Top

Summary of Refined Results for Additional Expermients

Back To Top

Observations of Refining Predicted Labels

Back To Top

Similar to the above, we see an improvement in quality of predicted labels once they are refined using a majority filter. Using this method, we end up achieving improved results in each of the experiments. However, Gear 4 is still usually poorly predicted and the additional experiment with validation skiers having high BMI is still the lowest performing experiment as this refining method works on the already predicted labels by claning them a bit to have more consistency during each set of gears observations.